Image capturing device including lens array and processing

ABSTRACT

Disclosed herein is an image pickup apparatus, including: an image pickup lens; a lens array disposed on an image formation plane of the image pickup lens; an image pickup device adapted to receive a light ray passing through the image pickup lens and the lens array to acquire picked up image data; and an image processing section adapted to carry out an image process for the picked up image data; the image processing section including a viewpoint image production section adapted to produce a plurality of viewpoint images based on the picked up image data, and an image synthesis processing section adapted to synthesize two or more viewpoint images from among the viewpoint images.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation of and claims the benefit under 35U.S.C. §120 of U.S. patent application Ser. No. 13/339,949, filed onDec. 29, 2011, which claims the benefit under 35 U.S.C. §119 of JapanesePatent Application JP 2011-001401, filed on Jan. 6, 2011. The entirecontents of these applications are hereby incorporated by referenceherein in their entireties.

BACKGROUND

The technology disclosed herein relates to an image pickup apparatus andan image processing method wherein a lens array is used.

Various image pickup apparatus have been proposed and developed and aredisclosed in, for example, PCT Patent Publication No. WO06/039486,Japanese Patent Publication No. Hei 5-76234, Japanese Patent PublicationNo. Hei 7-8055, and Japanese Patent No. 3182009 as well as Ren Ng, andseven others, “Light Field Photography with a Hand-Held PlenopticCamera,” Stanford Tech Report CTSR 2005-02. Also an image pickupapparatus in which a predetermined image process is carried out forpicked up image data and resulting data are outputted has been proposed.For example, in the first-mentioned patent document and the non-patentdocument mentioned above, an image pickup apparatus which uses a methodcalled “Light Field Photography” is disclosed. In the image pickupapparatus, a lens array is disposed on a focal plane of an image pickuplens and an image sensor is provided on a focal plane of the lens array.Consequently, it is possible for the image sensor to receive a video ofan image pickup object formed on the lens array while dividing the videointo rays of light in various viewpoint directions and acquiremulti-viewpoint images at the same time.

SUMMARY

In such an image pickup apparatus as described above, a plurality ofpixels on the image sensor are allocated to one lens in the lens array,in which the dimensions in the vertical and horizontal directions of thelenses are equal to integral multiples of a pixel pitch. Further, anumber of viewpoint images equal to the number of pixels allocated tothe lens can be acquired. For example, where one lens is allocated to3×3 pixels in which the vertical and horizontal dimensions of the lensare equal to pitches per three pixels, images at three viewpoints in thevertical direction and the horizontal direction can be acquired.Consequently, images per totaling nine viewpoints can be acquired. Bydisplaying, for example, two left and right viewpoint images from amongthe viewpoint images acquired in such a manner as described above usinga predetermined display unit, for example, a stereoscopic image displaycan be implemented.

However, upon stereoscopic image display, depending upon the displayedimage, the parallax amount between the left and right viewpoint imagesbecomes excessively great until it exceeds the stereoscopic perceptuallimitation of the human being. If the stereoscopic perceptual limitationis exceeded in this manner, then two viewpoint images are recognized asdouble images. In this instance, since there is the possibility that, ifsuch an image as described above is viewed continuously, then eyestrainor the like may be caused, implementation of a method for reducing theinfluence of such an excessively great parallax as described above onthe visibility is desired.

On the other hand, in Japanese Patent Publication No. Hei 5-76234mentioned hereinabove, a method is disclosed in which image pickup iscarried out in a state in which a diaphragm condition is changed foreach of picked up images to produce blurring, that is, to gradate acontour, to moderate the influence of the double images. However, in themethod, since the diaphragm condition is different for each of images,the method is less likely to be ready for a moving picture. Further, inJapanese Patent Publication No. Hei 7-8055, a technique is disclosed inwhich the picture quality of a defocused image of an image pickup objectof a stereoscopic video is degraded. Further, in Japanese Patent No.3182009, a process for calculating an average of pixels around a noticedpixel is carried out to form blurring.

Therefore, it is desirable to provide an image pickup apparatus by whicha viewpoint image with which good visibility is achieved uponstereoscopic image display can be acquired.

According to an embodiment of the disclosed technology, there isprovided an image pickup apparatus including an image pickup lens, alens array disposed on an image formation plane of the image pickuplens, an image pickup device adapted to receive a light ray passingthrough the image pickup lens and the lens array to acquire picked upimage data, and an image processing section adapted to carry out animage process for the picked up image data, the image processing sectionincluding a viewpoint image production section adapted to produce aplurality of viewpoint images based on the picked up image data, and animage synthesis processing section adapted to synthesize two or moreviewpoint images from among the viewpoint images.

In the image pickup apparatus, a light lay from an image pickup objectpassing through the image pickup lens is divided for individualviewpoint directions by the lens array and received by the image pickupdevice, by which picked up image data are obtained. The image processingsection produces a plurality of viewpoint images based on the obtainedpicked up image data and synthesizes two or more of the viewpointimages. The viewpoint image after the synthesis exhibits a state inwhich the position thereof is shifted and the viewpoint and a furtherdefocused state like an image having a blurred contour.

According to another embodiment of the disclosed technology, there isprovided an image processing method including acquiring picked up imagedata from an image pickup device which receives a light ray passingthrough an image pickup lens and a lens array disposed on an imageformation plane of the image pickup lens, producing a plurality ofviewpoint images based on the picked up image data, and synthesizing twoor more viewpoint images from among the plural viewpoint images.

With the image pickup apparatus, the image processing sectionsynthesizes two or more viewpoint images. Consequently, while theposition of a certain one of the viewpoint images is shifted, thecontour or the viewpoint image can be blurred. Therefore, for example,when two left and right viewpoint images after the synthesis are used tocarry out stereoscopic video display, such an excessively great parallaxas exceeds the perceptual limitation of the human being can be reducedto moderate the eyestrain and so forth by an effect by parallax amountsuppression and blurring of the viewpoint images. In other words, aviewpoint image which achieves good visibility upon stereoscopic videodisplay can be acquired.

In the image processing method, picked up image data are acquired fromthe image pickup device which receives a light ray passing through theimage pickup lens and the lens array disposed on the image formationplane of the image pickup lens. Then, a plurality of viewpoint imagesare produced based on the picked up image data, and two or moreviewpoint images from among the plural viewpoint images are synthesized.Consequently, for example, when two left and right viewpoint imagesafter the synthesis are used to carry out stereoscopic video display,such an excessively great parallax as exceeds the perceptual limitationof the human being can be reduced to moderate the eyestrain and so forthby an effect by parallax amount suppression and blurring of theviewpoint images. In other words, a viewpoint image which achieves goodvisibility upon stereoscopic video display can be acquired.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view showing a general configuration of an image pickupapparatus according to an embodiment of the technology disclosed herein;

FIG. 2 is a schematic view illustrating a disposition relationship of alens array and an image sensor;

FIG. 3 is a functional block diagram showing a detailed configuration ofan image processing section shown in FIG. 1;

FIG. 4 is a schematic view illustrating light ray division in viewpointdirections;

FIG. 5 is a schematic view illustrating picked up image data acquired bythe image pickup apparatus of FIG. 1;

FIGS. 6A to 6I are schematic views illustrating viewpoint image dataobtained from the image pickup data illustrated in FIG. 5;

FIGS. 7A to 7I are schematic views showing an example of viewpointimages corresponding to the viewpoint image data illustrated in FIGS. 6Ato 6I;

FIGS. 8A and 8B are schematic views showing a parallax amount betweenviewpoint images;

FIGS. 9A to 9C are schematic views illustrating image synthesisprocessing operation;

FIGS. 10A to 10C are schematic views illustrating appearances in thecase where a synthesis process is carried out for viewpoint imagesaccording to a comparative example 1;

FIGS. 11A to 11C are schematic views illustrating appearances in thecase where the synthesis process is carried out for viewpoint images bythe image pickup apparatus of FIG. 1;

FIGS. 12A to 12C are schematic views illustrating appearances in thecase where the ratio is changed to carry out the synthesis process forviewpoint images by the image pickup apparatus of FIG. 1;

FIG. 13 is a functional block diagram showing a detailed configurationof an image processing section according to a modification to the imagepickup apparatus of FIG. 1; and

FIGS. 14A to 14C are schematic views illustrating image synthesisprocessing operation by the image processing section shown in FIG. 13.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

In the following, an embodiment of the technology disclosed herein isdescribed in detail with reference to the accompanying drawings. It isto be noted that the disclosed technology is described in the followingorder:

1. Embodiment (example wherein a synthesis process of viewpoint imagesis carried out collectively on an image plane)

2. Modification (example wherein a synthesis process is carried out foreach selective region on an image plane in response to depthinformation)

Embodiment

General Configuration

FIG. 1 shows a general configuration of an image pickup apparatus 1according to an embodiment of the disclosed technology. The image pickupapparatus 1 is a monocular type camera for picking up an image using asingle image pickup lens. The image pickup apparatus 1 picks up an imageof an image pickup object 2, carries out a predetermined image processfor the picked up image pickup object 2 and outputs image data Dout as aviewpoint image. The image pickup apparatus 1 includes an image pickuplens 11, a lens array 12, an image sensor 13, an image processingsection 14, an image sensor driving section 15 and a control section 16.It is to be noted that, in the following description, an optical axis isrepresented by Z, and in a plane orthogonal to the optical axis Z, ahorizontal direction and a vertical direction are represented by X andY, respectively. Further, since the image processing method of thedisclosed technology is implemented by a configuration and operation ofthe image processing section 14, description of the image processingmethod is omitted herein.

The image pickup lens 11 is a main lens for picking up an image of animage pickup object 2 and is configured from a general image pickup lensused, for example, for a video camera or a still camera. A diaphragm 10is disposed on the light incoming side of the image pickup lens 11 butmay alternatively be disposed on the light outgoing side.

The lens array 12 is disposed on a focal plane or image formation planeof the image pickup lens 11 and is configured by two-dimensionallydisposing a plurality of lenses or micro lenses 12 a along the Xdirection and the Y direction on a substrate configured, for example,from glass. Each of the lenses 12 a is configured from a resin materialsuch as a photoresist and is formed using, for example, a resist reflowmethod or a nanoimprint method on a substrate configured from glass orplastic. Or, the lenses 12 a may be formed by carrying out an etchingprocess for the surface of the substrate. The image sensor 13 isdisposed on the focal plane of the lens array 12.

The image sensor 13 receives a ray of light passing through the lensarray 12 to acquire image pickup data D0. In the image sensor 13, aplurality of pixels are disposed in a matrix along the X and Ydirections. The image sensor 13 is configured from a solid-state imagepickup device such as a CCD (Charge Coupled Device) device or a CMOS(Complementary Metal-Oxide Semiconductor) device.

FIG. 2 schematically illustrates a disposition configuration of the lensarray 12 and the image sensor 13. Referring to FIG. 2, the lens array 12and the image sensor 13 are disposed in a spaced relationship from eachother by a predetermined distance, particularly by a focal distance ofthe lens array 12, along the optical axis Z such that m×n pixel regionsU in the image sensor 13 are allocated to one lens 12 a. The shape ofthe lens 12 a on an XY plane is a square shape, for example, same asthat of the m×n pixel region U. It is to be noted that m and n areintegers equal to or greater than 1, and as the value of m×n increases,that is, as the number of pixels to be applied to one microlensincreases, the viewpoint image number or number of visual pointsincreases. On the other hand, as the pixel number allocated to each lensdecreases, that is, as the value of m×n decreases, the pixel number orresolution of the viewpoint image increases. In this manner, the numberof visual points and the resolution of the viewpoint image have atradeoff relationship to each other. In the following, description isgiven taking a case in which a pixel region U of 3×3 pixels (m=n=3) isallocated to one lens 12 a as an example.

On a light receiving face of the image sensor 13, a color filter notshown may be provided. For the color filter, a color filter of the typecan be used wherein filters for the colors of, for example, red (R),green G and blue B are arrayed at a ratio of, for example, 1:2:1 asknown as Bayer array.

The image processing section 14 carries out a predetermined imageprocess for the picked up image data D0 acquired by the image sensor 13and outputs image data Dout, for example, as viewpoint images. FIG. 3shows a detailed configuration of the image processing section 14.Referring to FIG. 3, the image processing section 14 includes, forexample, a viewpoint image production section 140, an image synthesisprocessing section 141 and an image correction processing section 142.Particular image processing operation of the image processing section 14is hereinafter described.

The image sensor driving section 15 drives the image sensor 13 tocontrol exposure, readout and so forth of the image sensor 13.

The control section 16 controls operation of the image processingsection 14 and the image sensor driving section 15 and is configured,for example, from a microcomputer.

Working Effect

1. Acquisition of Picked Up Image Data

In the image pickup apparatus 1, since the lens array 12 is provided ata predetermined position between the image pickup lens 11 and the imagesensor 13, the image sensor 13 records a ray of light from an imagepickup object 2 as a light ray vector which retains informationregarding an advancing direction of the light ray, that is, a viewpointdirection, in addition to an intensity distribution of the light ray. Inparticular, a ray of light passing through the lens array 12 is dividedfor each viewpoint direction of the visual point and received bydifferent pixels of the image sensor 13. For example, as seen in FIG. 4,from within a ray of light passing through the lens 12 a, a light ray orlight flux LA from a direction of a certain viewpoint, that is, of afirst visual point, is received by a pixel “A.” Meanwhile, light rays orlight fluxes LB and LC from directions from visual points, that is, fromsecond and third visual points, different from the direction of thecertain visual point are received by pixels “B” and “C,” respectively.In this manner, in the pixel region U allocated to the lens 12 a, lightrays from different visual point directions from each other are receivedby different pixels. In the image sensor 13, readout is carried outline-sequentially in response to driving operation by the image sensordriving section 15, and picked up image data D0 are acquired.

FIG. 5 schematically illustrates a pixel data array of the picked upimage data D0. In the case where a pixel region U of 3×3 pixels isallocated to one lens 12 a as in the present embodiment, light rays fromtotaling nine visual point directions are received for each pixel regionU, and A to I pixel data of 3×3 pixels are acquired. It is to be notedthat, in FIG. 5, picked up image data obtained from 9×9 pixel regions of3×3 pixels, that is, from nine pixel regions U, are illustrated.Further, in the case where a color filter is disposed on the lightreceiving face side of the image sensor 13, the picked up image data D0are recorded as color data corresponding to the color array of the colorfilter. The picked up image data D0 obtained in such a manner asdescribed above are outputted to the image processing section 14.

2. Production of a Viewpoint Image

Referring to FIG. 3, the image processing section 14 includes aviewpoint image production section 140, an image synthesis processingsection 141 and an image correction processing section 142. The imageprocessing section 14 carries out a predetermined image process based onthe picked up image data D0 outputted from the image sensor 13 andoutputs image data Dout as a viewpoint image.

In particular, the viewpoint image production section 140 first carriesout a process of producing a plurality of viewpoint images based on thepicked up image data D0. In particular, the viewpoint image productionsection 140 synthesizes those of the picked up image data D0 illustratedin FIG. 5 which belong to the same visual point directions, that is,which are extracted from those pixels positioned at the same positionsof the pixel regions U. For example, the viewpoint image productionsection 140 extracts, from among the picked up image data D0, all pixeldata “A” and synthesizes the extracted pixel data as seen in FIG. 6A.Similar processing is carried out also for the other pixel data “B” to“I” as seen in FIGS. 6B to 6I. In this manner, the viewpoint imageproduction section 140 produces a plurality of viewpoint images, here,nine viewpoint images of the first to ninth visual points, based on thepicked up image data D0. The viewpoint images are outputted as viewpointimage data D1 to the image synthesis processing section 141.

3. Image Synthesis Process

The image synthesis processing section 141 carries out such an imagesynthesis process as described below for a plurality of viewpoint imagesof the viewpoint image data D1 inputted thereto.

Viewpoint images R1 to R9 shown in FIGS. 7A to 7I are a particularexample of viewpoint images corresponding to the data array of FIGS. 6Ato 6I. Here, an image of an image pickup object 2 is described takingimages Ra, Rb and Rc of three image pickup objects “person,” “mountain”and “flower” disposed at positions different from each other in thedepthwise direction are given as an example. The viewpoint images R1 toR9 are picked up such that the “person” from among the through imagepickup objects are focused by the image pickup lens, and the image Rb ofthe “mountain” positioned on the interior side with respect to the“person” and the image Rc of the “flower” positioned on this side withrespect to the “person” are in a defocused state. Since the viewpointimages R1 to R9 are picked up by a camera of a monocular camera having asingle image pickup lens, even if the visual point varies, the image Raof the “person” on the focus plane does not shift. However, thedefocused images Rb and Rc are shifted to positions different from eachother. It is to be noted that, in FIGS. 7A to 7I, the position shiftingbetween different viewpoint images, that is, the position shifting ofthe images Rb and Rc, is illustrated in an exaggerated form.

When, for example, two left and right viewpoint images from among suchnine viewpoint images R1 to R9 as described above are used to carry outstereoscopic image display, the stereoscopic effect of a video displayedcorresponds to the amount of parallax between the two viewpoint images.For example, if a viewpoint image R4 shown in FIG. 7D and a viewpointimage R6 shown in FIG. 7F are selected as the two viewpoint imagesdescribed above, then the stereoscopic effect of the display video issuch as described below. For example, although the “mountain” looks onthe interior side with respect to the “person,” the degree of suchlooking corresponds to the positional displacement amount or parallaxamount Wb between an image Rb4 in the viewpoint image R4 and an imageRb6 in the viewpoint image R6 as seen in FIGS. 8A and 8B. On the otherhand, although the “flower” looks protruding to this side with respectto the “person,” the degree of such looking corresponds to thepositional displacement amount Wc between an image Rc4 in the viewpointimage R4 and an image Rc6 in the viewpoint image R6. Then, as thepositional displacement amounts Wb and Wc increase, the “mountain” isobserved at a position by an increasing amount on the interior side andthe “flower” is observed at a position by an increasing amount on thisside.

However, if such parallax amounts are excessively great, then theyexceed the perceptual limitation of the human being, and the images lookas doubled images and are not recognized as stereoscopic videos. Or,even if the images are recognized as stereoscopic videos, the observeris likely to feel the eyestrain by continuously observing such a videoproximate to the perceptual limitation.

Therefore, in the following embodiment, in order to reduce theexcessively great parallax which causes such eyestrain as describedabove, a process of synthesizing or adding two or more viewpoint imagesat predetermined synthesis rates is carried out. The process mentionedis hereinafter referred to as synthesis process. Thus, a synthesisprocess of two or more viewpoint images is carried out by weighting theviewpoint images. This synthesis process can be represented by thefollowing expression (1):Rn=(α×R1+β×R2+γ×R3+δ×R4+ε×R5+ζ×R6+η×R7+θ×R8+ι×R9)  (1)where Rn is a viewpoint image after the synthesis process, and α, β, γ,δ, ε, ζ, η, θ and ι are coefficients representative of synthesis rates.The coefficients may be zero. In particular, all of the viewpoint imagesmay be synthesized or else they may be synthesized selectively. Further,such synthesis process can be carried out using any of plane data, linedata and pixel data. However, since, in the present embodiment, thesynthesis rates are equal among the image planes, the synthesis processis carried out collectively for the planes.Principle of Parallax Reduction

A principle of parallax reduction by such a synthesis process asdescribed above is described below with reference to FIGS. 9A to 12C.For the simplified description, an example wherein the viewpoint imageR4 and the viewpoint image R5 are to be synthesized and particularly, ofthe coefficients mentioned hereinabove, α, β, γ, ζ, η, θ and ι are zerowhile δ=1 and 0<ε<1 is described. FIGS. 9A to 9C represent a synthesisprocess of the viewpoint images as images. If the viewpoint image R4 andthe viewpoint image R5 multiplied by the coefficient ε are added, thenthe image positions of the “mountain” and the “flower” in the defocusedstate are shifted. In particular, if attention is paid to the “flower,”then the position of the “flower” is shifted to a position Sn between aposition S4 of the image Rc4 in the viewpoint image R4 and a position S5of the image Rc5 in the viewpoint image R5. Further, an image Rcn afterthe synthesis process has a contour rather blurred. This similarlyapplies also to the image of the “mountain,” and the position of animage Rbn of the same after the synthesis process is shifted to aposition between the positions of the images Rb4 and Rb5 and the imageRbn has a contour rather blurred as in a defocused state.

Incidentally, if viewpoint images picked up by a binocular camera areused to carry out a synthesis process similar to that described above,then it is difficult to blur an image itself while the image position isshifted as in the case illustrated in FIGS. 10A to 10C. FIGS. 10A to 10Cschematically illustrate signal intensity distributions in the casewhere viewpoint images picked up by a binocular camera are synthesizedas a comparative example with the present embodiment. In the comparativeexample, since an image itself is sharp also in a defocused region, inthe case where viewpoint images different from each other are added,doubled images are obtained. In particular, since the width or extendH100 of the contrast is small as seen in FIGS. 10A and 10B, in the casewhere contrasts A100 and B100 having peaks at positions S100 and S101different each other are synthesized, the contrast C100 after thesynthesis exhibits a distribution having two peaks at differentpositions S100 and S101 as seen in FIG. 10C. Further, in the case wherea camera of the binocular type is used, though not shown, since theposition of an image of an image pickup object on the focus plane isdifferent between the viewpoint images, it is difficult to carry outsuch a collective synthesis process for an entire image as in thepresent embodiment. In other words, a process for adjusting the imagepositions on the focus plane or a like process is required.

In contrast, although, in the present embodiment, it is possible tocarry out position shifting and contour blurring by such a synthesisprocess as described above, this arises from the following reason. FIGS.11A to 11C illustrate signal intensity distributions, that is, contrastsof contour portions of defocused images, in the case where viewpointimages acquired by the image pickup apparatus 1 are used to carry out asynthesis process. It is to be noted that, in the present example, thesynthesis rates of the individual viewpoint images are equal to eachother. Since, in the present embodiment, the width H of the contrast inthe defocused region is great, in the case where contrasts A and Bhaving a peak at places S4 and S5 which are different from each otherand corresponding to the images Rc4 and Rc5 of the viewpoint images R4and R5 are synthesized, a contrast C after the synthesis exhibits a moremoderate distribution without causing doubled images while the peaks atthe two positions disappear as seen in FIG. 11C. Accordingly, in thepresent embodiment, it is possible to blur the contour while theposition of such a defocused image is shifted by the synthesis process.Further, different from the viewpoint images picked up by a camera ofthe binocular type described above, the image position of an imagepickup object on the focus plane is same between the viewpoint images,and therefore, such a positioning process as in the case of thecomparative example is not required in the synthesis process. Inaddition, since only it is necessary to weight and add producedviewpoint images, also the processing load is low.

On the other hand, FIGS. 12A to 12C schematically illustrates signalintensity distributions in the case where the synthesis rate is madedifferent between viewpoint images in the synthesis process of theembodiment described above. Also in this instance, similarly as in thecase described above, when contrasts A and B1 having peaks at places S4and S5 different from each other and corresponding, for example, to theimages Rc4 and Rc5, respectively, are synthesized particularly in thecase where the (synthesis rate of R4)<(synthesis rate of R5), no doubledimages appear on a contrast C1 after the synthesis. Accordingly, thecontour can be blurred while the position of a defocused image isshifted. However, since the viewpoint images are individually weighted,the peak position Sn′ of the contrast C1 is displaced rather to theposition S5 side.

While the synthesis of two viewpoint images is described as an example,synthesis of three or more viewpoint images is carried out similarly.Further, the image position can be shifted freely within a predeterminedrange by making the synthesis rate different among the viewpoint images,that is, by weighting the viewpoint images. While, before the synthesisprocess, nine defocused images corresponding to the number of visualpoints are represented by different positions, by carrying out thesynthesis process, it is possible to shift the image to an arbitraryposition among the nine positions and blur the contour. Accordingly, anexcessively great parallax can be moderated by an effect of the parallaxamount suppression and an effect of blurring of an image.

Viewpoint images after such a synthesis process as described above areoutputted as viewpoint image data D2 to the image correction processingsection 142. The image correction processing section 142 carries out acolor interpolation process such as, for example, a demosaic process, awhite balance adjustment process, a gamma correction process and soforth for the viewpoint image data D2 and outputs the viewpoint imagedata after the image processes as image data Dout. The image data Doutmay be outputted to the outside of the image pickup apparatus 1 or maybe stored into a storage section not shown provided in the image pickupapparatus 1.

It is to be noted that the viewpoint image data D2 and the image dataDout may be only data corresponding to the viewpoint images produced bythe synthesis process or may be data corresponding to the originalviewpoint images, here the nine viewpoint images, for which thesynthesis process is not carried out. Or data corresponding to theviewpoint images before and after the synthesis process may be includedin a mixed state, or the original viewpoint images may be replaced bythe viewpoint images produced by the synthesis process. Further, thenumber of viewpoint images to be produced by the synthesis process maybe one or a plural number.

As described above, in the present embodiment, a plurality of viewpointimages are produced based on picked up image data D0 acquired using theimage pickup lens 11, lens array 12 and image sensor 13, and two or moreof the viewpoint images are synthesized. Consequently, while theposition of a certain viewpoint image is shifted, the contour of theviewpoint image can be blurred. Therefore, for example, when two leftand right viewpoint images are used to carry out stereoscopic imagedisplay, such an excessively great parallax as exceeds the perceptuallimitation of the human being can be reduced to moderate the eyestrainand so forth by an effect provided by parallax amount suppression andblurring with regard to the viewpoint images. In other words, uponstereoscopic image display, viewpoint images which have good visibilitycan be acquired.

In the following, a modification to the embodiment described above isdescribed.

<Modifications>

FIG. 13 shows an example of a detailed configuration of an imageprocessing section 14A according to a modification to the embodiment.Referring to FIG. 13, similarly to the image processing section 14 ofthe embodiment described hereinabove, the image processing section 14Acarries out a predetermined image process for picked up image data D0acquired by the image pickup lens 11, lens array 12 and image sensor 13and outputs image data Dout. However, in the present modification, theimage processing section 14 includes a depth information acquisitionsection 143 in addition to the viewpoint image production section 140,image synthesis processing section 141 and image correction processingsection 142. The image synthesis processing section 141 carries out asynthesis in response to depth information.

In particular, in the image processing section 14A, the viewpoint imageproduction section 140 first produces a plurality of viewpoint imagessimilarly as in the first embodiment described hereinabove based onpicked up image data D0. For example, the viewpoint image productionsection 140 produces viewpoint images R1 to R9 of first to ninth visualpoints and outputs viewpoint image data D1.

Meanwhile, the depth information acquisition section 143 acquires depthinformation, for example, information representative of the depth of animage pickup object such as a disparity map, from the picked up imagedata D0. In particular, the depth information acquisition section 143calculates a disparity, that is, a phase difference or phasedisplacement, between a plurality of viewpoint images for each pixel,for example, by a stereo matching method, and produces a map wherein thecalculated disparities are associated with the individual pixels. It isto be noted that the disparity map is not such a disparity map producedbased on a unit of a pixel as described above but may be a disparity mapwherein disparities determined for individual pixel blocks each formedfrom a predetermined number of pixels are associated with the pixelblocks. The produced disparity map is outputted as depth information DDto the image synthesis processing section 141.

Then, the image synthesis processing section 141 uses the viewpointimages R1 to R9 and the depth information DD inputted thereto to carryout a synthesis process. In particular, the image synthesis processingsection 141 weights two or more of the viewpoint images R1 to R9 foreach selective region in the image plane in response to the depthinformation DD and then carries out a synthesis process. Moreparticularly, the image synthesis processing section 141 sets acomparatively low synthesis rate to an image of an image pickup objecton the image plane positioned comparatively near to the focus plane butsets a comparatively high synthesis rate to an image of another imagepickup object spaced comparatively far from the focus plane to carry outa synthesis process for each selective region.

FIGS. 14A to 14C illustrate an example of an image synthesis process inthe present modification. It is to be noted here that, for thesimplified illustration and description, an image Ra of a “person” onthe focus plane and images Rc41 and Rc42 of two “flowers” positioned onthis side with respect to the “person” are shown in the figures.Further, from between the images Rc41 and Rc42, the image Rc42 isobserved on this side with respect to the image Rc41. Or in other words,the image Rc42 is in a state more defocused than the image Rc41 and isan image of the “flower” displaced by a greater amount from the focusplane. In such an instance as just described, for example, when theviewpoint image R4 and the viewpoint image R5 are to be synthesized,synthesis processes for the images Rc41 and Rc42 as represented by thefollowing expressions (2) and (3), respectively, are carried out:Rc41+ε1×Rc51=Rcn1  (2)Rc42+ε2×Rc52=Rcn2  (3)where the synthesis rates satisfy ε1<ε2.

Consequently, as seen in FIG. 14C, in a viewpoint image Rn after thesynthesis process, the position of the image Rcn1 of the “flower” isshifted to a position between the positions of the images Rc41 and Rc51and the contour of the image Rcn1 is blurred from the principledescribed hereinabove with reference to FIGS. 11A to 11C. Also theposition of the image Rcn2 of the “flower” is shifted to a positionbetween the positions of the images Rc42 and Rc52 and the contour of theimage Rcn2 is blurred similarly. However, since the weighting is madedifferent between the images Rc41 and Rc42 (E1<E2), the shift amount ofthe image Rcn2 is greater than that of the image Rcn1 from the principledescribed hereinabove with reference to FIGS. 12A to 12C. Moreparticularly, the degree by which the position of the image Rcn2approaches the position of the image Rc52 is higher than the degree bywhich the position of the image Rcn1 approaches the position of theimage Rc51. Consequently, although an image spaced by a greater distancefrom the focus plane, that is, an image defocused by a greater amount,exhibits a greater parallax amount than the other viewpoint image, theshift amount can be increased for such a defocused image as describedabove. In other words, the contour of the image can be blurred while theparallax amount is suppressed in response to the parallax amount on theimage plane. Consequently, for example, such a synthesis process can becarried out that, with regard to an image whose parallax amount is notvery great, phase shifting or contour blurring is not carried out whileposition shifting and contour blurring are carried out only with regardto an image having a great parallax amount.

A plurality of viewpoint images after such synthesis process areoutputted as viewpoint image data D2 to the image correction processingsection 142 and are subjected to an image correction process similar tothat described hereinabove and then outputted as image data Dout.

The image correction processing section 142 may carry out a synthesisprocess for each selective region on the image plane based on depthinformation as in the modification. Also in such an instance, a similareffect to that achieved by the embodiment described hereinabove can beachieved. Further, by using depth information, with regard to images ofimage pickup objects which are different in depth or parallax amount onthe image plane, the image contour can be blurred while parallaxsuppression in accordance with the parallax amount is carried out.Therefore, more natural stereoscopic video image display can beimplemented.

While the disclosed technology is described above in connection with thepreferred embodiment and the modification, the technology is not limitedto the embodiment and the modification, but further variousmodifications are possible. For example, while, in the embodimentdescribed above, the number or region of lens allocation pixels ism×n=3×3 pixels, the pixel region to be allocated to each lens is notlimited to this, but, for example, m and n may be 1, 2 or 4 or more.

Further, while, in the embodiment described above, a disparity map isproduced as depth information and is used to carry out an imagesynthesis process, the depth information is not limited to such adisparity map but may be information acquired by any other method. Forexample, since the resolution of an image of an image pickup objectdiffers depending upon the distance from the focus plane, the depthinformation of each image may be acquired based on the resolution. Moreparticularly, since the image of the “person” on the focus planeexhibits a high resolution while the “mountain” or the “flower” isspaced from the focus plane, the image of the same is in a defocusedstate and exhibits a low resolution.

Further, in the embodiment and the modification described above, in anexample of the image processing method of the disclosed technology, thatis, in the image processing method realized by the image processingsection 14, one image pickup apparatus including an image pickup lens, alens array and an image sensor acquires picked up image data and carriesout a predetermined image process based on the picked up image data.However, the picked up image data which make an object of the imageprocess need not necessarily be acquired by one apparatus. In otherwords, picked up image data corresponding to a plurality of viewpointimages may be acquired from the outside, or a predetermined imageprocess may be carried out for picked up image data acquired from theoutside.

The present disclosure contains subject matter related to that disclosedin Japanese Priority Patent Application JP 2011-001401 filed in theJapan Patent Office on Jan. 6, 2011, the entire content of which ishereby incorporated by reference.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors in so far as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. An image processing apparatus, comprising: one ormore processors configured to: receive a plurality of viewpoint imagesbased on picked up image data acquired on the basis of a light raypassing through an image pickup lens and a lens array; carry outweighting for each of the plurality of viewpoint images in an imageplane based on depth information; and synthesize two or more viewpointimages from the plurality of viewpoint images by using said weighting.2. The image processing apparatus of claim 1, wherein the one or moreprocessors are configured to synthesize the two or more viewpoint imagesfor each of two or more respective region blocks in the image plane. 3.The image processing apparatus of claim 2, further including: the imagepickup lens; the lens array disposed on an image formation plane of theimage pickup lens; and an image pickup device configured to receive thelight ray passing through the image pickup lens and the lens array toacquire the picked up image data.
 4. An image processing method,comprising: in one or more processors: receiving a plurality ofviewpoint images based on picked up image data acquired on the basis ofa light ray passing through an image pickup lens and a lens array;performing weighting for each of the plurality of viewpoint images in animage plane based, at least in part, on depth information andsynthesizing two or more viewpoint images from the plurality ofviewpoint images by using said weighting.
 5. The image processing methodof claim 4, wherein the plurality of viewpoint images correspond to afirst region in the image plane, and wherein the method furthercomprises producing one or more fourth viewpoint images by synthesizingtwo or more third viewpoint images corresponding to a second region inthe image plane.
 6. The image processing method of claim 5, wherein thelens array is disposed in a focal plane of the image pickup lens.
 7. Animage processing apparatus, comprising: one or more processorsconfigured to: perform weighting of two or more first viewpoint imagesbased, at least in part, on depth information, the two or more firstviewpoint images corresponding to an image plane and being based onpicked up image data, and produce one or more second viewpoint images bysynthesizing the weighted two or more first viewpoint images, whereinthe picked up image data is acquired on the basis of a light ray passingthrough an image pickup lens and a lens array.
 8. The image processingapparatus of claim 7, wherein the two or more first viewpoint imagescorrespond to a first region in the image plane, and wherein the one ormore processors are configured to produce one or more fourth viewpointimages by synthesizing two or more third viewpoint images correspondingto a second region in the image plane.
 9. The image processing apparatusof claim 8, further including: the image pickup lens; the lens arraydisposed in a focal plane of the image pickup lens; and an image pickupdevice configured to receive the light passing through the image pickuplens and the lens array to acquire the picked up image data.